The VideoToolbox allows you to produce accurately controlled visual stiouli. (David Brainard's comment about the VideoToolbox is that it allows you to ignore the fancy graphics model embodied in Macintosh QuickDraw, reducing your Macintosh to a plain frame store.) This document deals with time; how to make sure each video frame shows what you want. In addition to reading this document (and the accompanying list of "Video bugs") you'll want to run the program TimeVideo, which gives all your video cards a thorough workout and reports all their timing characteristics. If you have many computers, put TimeVideo on a floppy and run it from the floppy on each machine; all the results will accumulate in a single report file. I am always glad to receive copies of the TimeVideo report on new machines.
MOVIES. It's easy to show movies on a Mac. Create all your GWorlds in memory ahead of time, and then call CopyWindows() to copy one image after another to the screen (e.g. try the demos NoiseVBL or SandStorm). For any serious application you'll want a real-time movie, one image per video frame, so you'll want to wait for a new frame before copying each image, using one of the synchronization techniques described below (e.g. try the demo NoiseVBL). How big a movie you can show in real time will depend on how fast your processor is. TimeVideo does this timing for you, telling you what fraction of the screen you can fill with a real-time movie. If you can't do a full-screen movie and you need to show multiple patches, consider showing several small movies, updating only the dynamic parts of the screen. (We assume that you are not limited by memory, and therefore do not discuss Apple's QuickTime, whose principal purpose, beyond making movies portable, is to minimize storage requirements by image compression.) Most Macs access built-in video much more quickly than NuBus video. Here are data rates for showing movies on a few machines:
Power Mac 8100/80: Built-in VRAM 18 MB/s, DRAM 17 MB/s
Power Mac 7100/66 (w secondary cache??): Built-In VRAM 29 MB/s
Power Mac 6100/60 (w secondary cache): Built-in VRAM 12 MB/s, DRAM 23 MB/s
Power Mac 6100/60 (w/o secondary cache): Built-in VRAM 11 MB/s, DRAM 12 MB/s
Quadra 840av: Built-in 12 MB/s, NuBus 8 MB/s
Quadra 630: Built-in 12 MB/s
Quadra 950: Built-in 11 MB/s, NuBus 5 MB/s
LC 475: Built-in 10.6 MB/s
Centris 660AV: Built-in 10.2 MB/s
Quadra 660AV: Built-in 9.6 MB/s
Quadra 900: Built-in 9.5 MB/s, NuBus 5.5 MB/s
Centris 650: Built-in 5.8 MB/s
LC III: Built-in 5.8 MB/s
IIci: Built-in 5.5 MB/s, NuBus 3.7 MB/s
II with Rocket 68040 accelerator: NuBus 4.0 MB/s
IIfx: NuBus 3.0 MB/s (8•24GC "acceleration" reduces it 1.5 MB/s)
PowerBook 160: Built-in LCD screen and video port 3.0 MB/s
The quoted rate is the best, across all pixel depths and across CopyBits and CopyBitsQuickly, which are usually within +/-20% of each other. CopyBits, being part of QuickDraw, always runs native on the PowerPC. Surprisingly, native speed for CopyBitsQuickly is only about 10% higher than emulated speed on the PowerPC. However, CopyBitsQuickly is an unusual program, with unrolled loops and other tricks to maximize speed. On the other hand, note the remarkable doubling of DRAM video speed (and 10% increase of VRAM speed) due to installation of the secondary cache. QUERY: Does anyone know how TimeVideo might determine whether a secondary cache is installed?
Power Mac 7100 NuBus bug: Michael Eckert (meckert@eniac.seas.upenn.edu) writes, "From what I understand, a bug in the 7100's NuBus interface limits the throughput of the Nubus port. This does not affect built-in VRAM video or DRAM video (the AV option, in PDS slot). This bug occurs only in the 7100 (not 8100), and is supposed to be fixed during the next release of the machines." October 12, 1994.
SPRITES. If you want to show objects moving around the screen, and your computer isn't fast enough to do a full screen movie, then you may want to animate just the objects. Objects animated in this way are called "sprites". Tony Myles has published a package called "SpriteWorld" that you may want to download from info-mac/Development. (For an explanation of how to do that, read the VideoToolbox "Advice" document.)
PAGE SWITCHING. Many video cards have multiple video pages; check your TimeVideo report, or call GDGetPageCnt(). Normally the Macintosh only uses page 0, but if you hawe multiple pages, they can be switched by calling GDSetPageShown() and GDSetPageDrawn(). This could be used to show very short full-screen movies, e.g. alternating two images. This might be an effective way to show large slow movies, taking several frames to load each new image to an unseen page that's instantly swapped in when it's full. (The number of pages available depends on the pixel size. In max depth there's always only one page. In lesser depths there's unused memory, which the video driver may make available as extra pages.) Most of the VideoToolbox is throughly tested, but my page switching routines have never been used. Beware.
CLUT (Color Lookup Table) ANIMATION. Most video cards have a hardware color lookup table (clut) that dynamically transforms each pixel to a color (an RGB triplet) that is sent to the digital to analog converters. Temporal modulation (e.g. flicker or fading on and off) of visual stimuli can be achieved very conveniently by loading a new clut on each frame. Look at the demo FlickeringGrating. However, not all video card drivers are fast enough to achieve this. (Usually this is not a hardware limitation; it's crumby driver software.) Run TimeVideo to find out for sure; it determines how many frames it takes either GDSetEntries or SetEntriesQuickly to load the clut. The clut is also used for gamma correction. Apple's provision for gamma correction is somewhat crude; for the utmost luminance accuracy you may wish to use the Luminance.c routines. See section B, below.
MAKING DYNAMIC WHITE NOISE. I know of at least three people (including myself) that have hit upon clut animation as a fast way to synthesize dynamic white noise. You fill the image plane with random numbers in the range 0 to 255, and you load a new random clut on each frame. It might seem that this will produce noise that is spatially and temporally uncorrelated. Not quite. Any pair of pixels has a 1/256 chance of being driven by the same clut entry, in which case they will be perfectly correlated. So the spatial autocorrelation function has the (desired) delta function at zero separation and an (undesired) 1/256 correlation at all other separations. This corresponds to the desired flat spectrum, PLUS an undesired delta function at zero spatial frequency. In plain English, you'll end up with dynamic white noise plus a spatially-uniform temporally-white flicker. (This problem does not arise if you only use each clut entry once, but that restricts you to an image of only 16x16.) Note that I am only discussing first and second order statistics. Higher-order statistics will still differ from truly white noise, but I doubt that the visual system is sensitive to the difference. The spatially uniform flicker component of the clut animation noise is visually very salient. Ted Adelson noticed it immediately when I proudly demonstrated this method at Optical Society in San Francisco in 1989 (I think). The artifactual flicker becomes progressively more salient as you increase the viewing distance, since the white noise eventually disappears and the spatially uniform flicker is unaffected. What I now do, and recommend to others, is to use brute force. Precompute all the noise and show a movie where all the pixels are truly independent random samples. The VideoToolbox routine RandFill() is a fast way to compute the noise, and is used by MakeNoise.c.
FANCIER TRICKS. The video card has only one clut, but it is possible to simultaneously present multiple independently modulated patterns, by reserving separate sections of the clut for each temporal modulation, though you'll suffer a comensurate loss of intensity resolution. Clut animation is not restricted to dynamic modulation of contrast. If the pixel values represent phase, and the clut is loaded with a sin function, then rotating the clut entries--i.e. setting the i-th entry to what was formerly the value of the i-1 entry--will shift the phase of the pattern. A spatially vignetted drifting grating may be synthesized by using alternate pixels (or frames) to represent vigoetted sin aod cos components of the pattern, devoting half the clut to each (or loading alternate cluts on alternate frames). Appropriate adjustment of the relative contrasts of the sin and cos components will produce aoy desired phase of the sum. However, most of these tricks are superfluous; it’s usually easier and better to just show a movie. Many current processors are fast enough to show 8-bit full-screen movies.
A. HOW TO SYNCHRONIZE A MACINTOSH PROGRAM TO A VIDEO CARD.
The hardest part in doing vision experiments is synchronizing the computer program to the video card. Nearly all video cards are masters, and run freely, expecting the video monitor to be a slave. Any experiment that cares about timing will normally have to synchronize itself to the video card, by waiting for each frame to end, as will be discussed below.
You typically can't access the video card hardware directly, because it's undocumented. Instead you are allowed to send requests to the video driver associated with your video card. All video cards that plug into the Mac come with video drivers that conform sufficiently well to the Apple guidelines that they are at least minimally compatible with the VideoToolbox. However, some drivers have outright bugs, and the timing of many drivers, for which Apple does not publish guidelines, sometimes makes it surprisingly difficult to do simple things. A current list of these bugs and "features" appears in the VideoToolbox "Video driver bugs" document.
Every video card has a video driver in its ROM, but the manufacturer may supply a newer driver by floppy that supercedes the one in ROM, which may be buggy. (Built-in video devices, e.g/ in the Mac IIci and Quadra computers, load the video driver from the computer's ROM.) Apple distributes video driver updates as resources in the System file, so updating your System may change the video driver of your Apple video card. IdentifyVideo(device) returns a string with the name and version number of the driver that is actually in use. Try TimeVideo.
There are three different ways to synchronize a program to a video card, which we'll first describe quickly, and then more fully. (1) Apple recommends using the vertical blanking interrupts, which supposedly occur once per frame, at the beginning of the vertical blanking. The VideoToolbox VBLInstall.c routines make this vgry easy to do. A disadvantage of this approach is that it fails if you raise the processor priority to block all interrupts. (2) Most (not all) video drivers, when asked to load the clut, wait until the vertical blanking interval before beginning to load the clut. (They wait in order to avoid creating visible hash on the screen while the clut is changing.) This has the side effect of synchronizing your program to the display, since the driver doesn't return control until the VBL interval occurs. Either of these synchronization methods (1 or 2) may fail, depending on which video card you have, what you've set the pixel size to (1 to 32 bits), and whether you've raised the processor priority. The demo program TimeVideo tests both methods of synchronization on all your video cards at all pixel sizes and saves the results in a text file. (3) I think that all video cards provide a read-only vertical-blanking bit, but unfortunately few manufacturers will tell you its address, so you have to find it yourself (typically by disassembling the video driver), which is tedious.
1. VBL Interrupts. Most video cards emit one VBL interrupt during each video frame. However, during 1990-1 Apple generated some poor video drivers for their cards ("Macintosh Display Card": 8•24) and the built-in video in the Quadra 700 and 950. These drivers generate several interrupts per frame. (Kyle Cave discovered that there are no extra interrupts if the cache is disabled on the Quadra 700.) This behavior is contrary to Apple's documentation, but VBLInstall.c works around the bug, using a timer to discard the excess interrupts (as suggested by Raynald Comtois). Some video drivers take multiple frames to load the clut, and block interrupts while doing so, wreaking havoc with any attempt to count frames. Apple's video drivers produced before 1990 and since 1992 seem to be ok. (This may be in response to my many bug reports.) Try the demos TimeVideo and NoiseVBL.
The best current theory on the source of the extra interrupts (due to Raynald Comtois) is that the card is asserting the interrupt line for a fixed duration, perhaps 1 ms. As long as you are in the interrupt-handling routine (or one of higher priority), other interrupts of the same level are disabled. If you stay in the interrupt routine for long enough, the interrupt line has time to be released and you don't get a second interrupt, otherwise you do.
2. LOADING THE CLUT. It seems that all video drivers automatically synchronize clut-loading to the video frame. Apple's guidelines for video drivers (in the book Designing Cards and Drivers) state that the video driver may save the clut-load (i.e. set-entries) request and defer the actual loading until the next VBL interrupt. Such video drivers return almost immediately after a set-entries request. Thus, although the actual clut load will be synchronized to the display, your program will be asynchronous because the GDSetEntries call will return immediately. However, Apple's guidelines state that if the processor's interrupt priority has been raised, suspending the video card's VBL interrupts, then the video driver should always load the clut before returning, which would synchronize your program to the display. You can raise and lower the processor priority by calling the VideoToolbox routine SetPriority.c. The priority is normally zero. TimeVideo measures timing at both normal and high psiority.
Be aware that a set-entries request, i.e. calling GDSetEntries(), does not necessarily wait for the beginning of the next vertical blanking interval. It might merely wait until blanking is true. As a result two successive GDSetEntries calls might occur during the same blanking interval. To get exactly one call per frame you may need to delay for a suitable interval (perhaps 1 ms) before calling it again.
Most video drivers that I've tested seem to be synchronous and reliably take exactly one frame to load the clut. However, some video drivers take longes, e.g. any loading of the clut on Apple's new video cards ("Macintosh Display Card": 8•24) takes 30 ms--two frames--which is unacceptably slow for lookup table animation. The TrueVision NuVista seems to be asynchronous. Presumably the on-board processor accepts the clut information at any time and actually updates the clut during the next vertical blanking interval. However, this means that asking the video driver to load the clut doesn't have the useful side effect of synchronizing the Mac program to the display. The NuVista takes about 0.3 frames (i.e. a few ms) to reload the whole clut when in 8, 16, or 32-bit mode, but takes several frames (i.e. tens of ms) when in 1, 2, or 4-bit mode (presumably because the fractional byte addressing is slow). I haven't checked, but presumably the NuVista driver would become synchronous, as specified by Apple, if the processor priority were raised.
SetEntriesQuickly.c--written primarily by Raynald Comtois, Peter Lennie, and Bill Haake--provides fast clut loading for many popular video devices. Try TimeVideo.
If 8-bit pixels are enough, consider buying Apple's old "Toby" or "TFB" video cards, since they work fine, and cost only $90 each from Shreve Systems (800)-227-3971.
3. BLANKING BIT. Apparently all video cards have a blanking bit (telling whether the video signal is in the blank period between frames), but, alas, the manufactwrers never tell you where it is. You could figure out which bit by disassembling the driver, as I did for the Apple Toby and TFB video cards (no longer sold by Apple; call Shreve at phone number given above). Perhaps one could write a program that would find the VBL bit automatically, by looking for any bit that changes at the right frequency.
WaitForBlanking() in SetEntriesQuickly.c tests the blanking bit, but presently supports only the (obsolete) Toby and TFB video cards. Hopefully other people will enhance this routine to support more devices.
4. CONCLUSIONS. Don't take synchronization or lookup table animation for granted. Run TimeVideo to check out your video cards.
B. SYNCHING MULTIPLE VIDEO CARDS TO EACH OTHER
SOFTWARE: Short of butchering the hardware, synching two video cards is hard to do. It needn't be so, if the manufacturers were more forthcoming about how to program their cards. By disassembling the video driver for Apple's original "Toby" video cards I worked out that there are halt and restart commands that can be issued to the card. Several video cards can be synched by restarting them all at once (actually one after the other, but this is quick). Their quartz crystals run at slightly different frequencies so the cards slowly drift out of phase with each other, but that's still good enough for a relatively brief visual stimulus, resynching before each stimulus. Software to do this for the Toby video cards (which are still available from Shreve Systems (800)-227-3971, $90 each) is in the VideoTFB.c file in the VideoToolbox. It seems very likely that this approach is applicable to most video cards, but you would need to discover what the appropriate commands are to restart your cards. (Normally, video cards are restarted only at restart.) You might be able to do this by disassembling the video drivers, but perhaps the manufacturer would tell you if you asked very nicely. (Incidentally, the "Toby" video cards also have an undocumented ribbon cable connector that an Apple engineer explained to me could be used to achieve hardware-level synchronization. He said it might not work. I tried but never got it to work.) I hope you'll share any useful results with me. I'd love to learn the restart commands for other video cards, especially Apple's, and would add support for them to the VideoToolbox software.
HARDWARE: David Brainard <brainard@condor.psych.ucsb.edu> writes, "We have been working on the problem of synchronizing two video cards. For the Apple 8•24 card (rev B), we have a simple hardward solution. It turns out you can cut the clock trace and push the clock signal through a fast OR gate. Somewhat to my surprise, you can hold the clock for as long as you like without changing the state of the board. When you release the clock, the sync is shifted by the amount of time you held the clock. The 8•24 cards are OK. They will drive a 16" monitor and SetEntriesQuickly works fast. They cost $239 used. There may be a software solution like the one you developed for the Toby cards, but my guy struggled with it for a while and gave up. (He is better with hardware than software, which is why we went the way we did.)"
BB. MORE ON INTERRUPTS
Michael Bach (bach@sun1.ruf.uni-freiburg.de) writes, "We are trying to do simultaneous visual stimulation & analog measurement of brain potentials on the same Macintosh. One of our ideas was to use the time manager to trigger analog measurement every 2 ms in its interrupt service routine. This works fine unless we use animate palette (or GDSetEntries or GDSetGamma): No timer tasks are serviced until vbl. It looks as if the
video driver goes into a tight loop, polling a vbl flag, with interrupts switched off! Can this be true? This would be a strong setback for using the Mac for this sort of work." Yes, alas, the normal behavior for interrupt service routines and the driver Control calls on the Mac is to suspend all interrupts until returning. Most video card drivers wait for vbl, as Michael says. However, TrueVision's NuVista returns immediately, letting the video card processor do the work later, at vbl time. And SetEntriesQuickly.c, for the few devices that it supports, allows you to achieve whatever behavior you want. Those solutions will make your code highly dependent on the particular video card. One solution that might work on all video cards would be to only call the video driver (e.g. GDSetEnries) during vertical blanking, i.e. immediately after the vbl interrupt occurs (use VBLInstall.c).
C. CONTROLLING THE COLOR LOOKUP TABLE (CLUT)
QuickDraw is one of the great virtues of the Macintosh. However, several of its assumptions about what you want are inappropriate for vision experiments. In particular, it assumes that you want all your monitors to act as one consistent desktop (with consistent color tables). This is a problem if you want to load completely independent lookup tables and images onto two monitors that, for example, you may want to superimpose optically. QuickDraw enforces the consistency throught the Palette Manager, but calls that are nominally to the Color Manager (e.g. Apple's SetEntries) may be intercepted by the Palette Manager, resulting in undesired effects on other screens. My solution is to bypass QuickDraw and to load the lookup tables more or less directly, without telling QuickDraw.
GDSetEntries() and SetEntriesQuickly() work outside of QuickDraw. GDSetEntries uses the video driver; SetEntriesQuickly goes directly to the hardware. They load the clut of the video card without changing the color spec table of the graphics device; QuickDraw will continue to assume that the graphics device's color spec table is a true copy of the clut. This is the behavior that I usually want. However, it may cause problems if you use CopyBits since CopyBits will translate the color of each pixel it copies, using the inverse color table of the current device, which is based on the color spec table, NOT the clut. If you want to use CopyBits or SetCPixel, you should copy your color spec table into the graphics device's color spec table, and set ctSeed to alert QuickDraw that it's been changed. Or, instead of using QuickDraw's CopyBits and SetCPixel, you could use the VideoToolbox's CopyBitsQuickly and SetPixelsQuickly(), which copy pixel walues directly, ignoring all color tables and inverse color tables, .
D. UNUSUAL FRAME RATES
For some vision experiments it is desirable to run at very high frame rates, e.g. 100 or even 200 Hz for retinal physiology experiments. There exist special monitors, e.g. the Joyce DM-4 Display, that run at such high frame rates, but no video cards automatically configure themselves to work with such odd displays. All Mac video cards that I know of are programmable to a wide range of line and frame rates, but very few video card manufacturers provide information on how to do this programming. (Normally, the video card senses the kind of monitor through the monitor sense lines and the video driver programs the card to suit the monitor. Alas, the monitor sense line scheme basically only identifies the Apple monitors, and is not a general purpose solution. The video driver is a program that is read from your video card's ROM into memory when you boot the computer.) Truevision does supply information on how to program the NuVista to arbitrary line and frame rates. (I believe the max frame rate is over 100 Hz, but I don't recall what the limit is.)
E. MORE-THAN-8-BIT DACs
RasterOps “ProColor 32” has 9-bit DACs.
RasterOps “Paintboard Turbo XL” has 9-bit DACs. $1399 (Mentioned by Steve Shevell)
Does anyone know of any others?
F. SHIELDCURSOR
From Steve Lemke at Radius (lemke@radius.com) (summer '93):
"Apple has stated several times (though I wouldn't necessarily expect you to have run across any of them) that if you write directly to the screen, you must use the "ShieldCursor" and "ShowCursor" traps to prevent the cursor from being overwritten. To do so, you pass the rectangle in which you are drawing to "ShieldCursor". It's nice that all QuickDraw routines use this trap, as it makes for a simple way to identify what areas of the screen are changing. It also makes for a simple way for people who draw directly to the screen to notify the Radius PowerView software that they are doing so. Everything in that rectangle gets updated on the Radius PowerView screen. Too bad more programs don't use it, though..."
Taking his advice, CopyWindows calls ShieldCursor before calling CopyBitsQuickly.
G. DISASSEMBLING A VIDEO DRIVER
Assuming you can read 680x0 assembly code, use the VideoToolbox utility GrabVideoDrivers to put the driver into a file, then use ResEdit with the ResEdit CODE editor to examine it, comparing it with the example in the appeodix of Apple's Designing Cards and Drivers book. The ResEdit CODE editor is a public domain file distributed by:
(The following remedies worked for the Mac IIci video driver, patching or replacing the buggy version 0 .Display_Video_Apple_RBV1driver by copying the bug-free version 1 of the same driver from the Mac IIsi. It is very likely that an analogous approach could be used to patch or replace the buggy version 0, 1, and 2 .Display_Video_Apple_DAFB drivers in the Quadra 700, 750, and 900, by the bug-free version 3 or 5 of that driver in the Centris 650 or the LC 475.)
The Mac IIci built-in video driver (.Display_Video_Apple_RBV1 driver, version 0) has a bug that causes it to crash if you try to do a getEntries Status request. Here are two ways to fix the bug:
1. AUTOMATIC TEMPORARY PATCH. (This happens automatically if you use GDVideo. This note is an explanation, in case you're curious.) The bug only affects GDGetEntries, so the first time GDGetEntries is invoked it automatically calls PatchMacIIciVideoDriver() in GDVideo.c to find and patch the copy of the buggy driver residing in memory, preventing any trouble. (If the driver is in ROM, then the driver is copied to RAM, and patched there.) Only two instructions are modified, to save & restore more registers. And the driver's version number is changed from 0 to 100, so that programs can distinguish it from the buggy version 0. This fix persists only until the next reboot.
2. PERMANENT UPGRADE. The Mac IIsi has version 1 of the same driver, without the bug. In principle all you have to do is copy the new driver from the IIsi to your IIci, but this is nontrivial because these drivers are in ROM. It is fairly simple, as described below, to place a copy of the version 1 (i.e. fixed) driver into the System file of the Mac IIci, but, as explained in Inside Mac V-424, this will only displace the older version 0 (buggy) driver if it's not driving the boot monitor (i.e. if the "Welcome to Macintosh" message appears on some other monitor), because the boot monitor's driver is loaded at boot time before the System file is available. If you have multiple monitors, you can use the Monitors Control Panel to change the boot monitor: hold the option key down and drag the smiling Mac to any monitor except the one driven by the built-in video. So, in case you do have multiple monitors, here's how to copy the driver. First put the VideoToolbox utility "GetVideoDrivers" on a floppy and run it on a Mac IIsi. That will copy the all the video drivers onto your floppy as ResEdit files. Then put the floppy in your Mac IIci and copy "-Display_Video_Apple_RBV1" onto your hard disk. Make a copy of your System file. Open the "-Display_Video_Apple_RBV1" file in ResEdit. Hit Command-I to edit the Resource Info. Set the "System Heap" and "Purgeable" flags, change the ID from 1 to 122, and close the Resource Info window. Now copy the resource and paste it into the copy of your System file. Quit, saving changes. Now reoame the active System file to something else, like "old System", and rename the edited System file to "System". Reboot. You can now throw the old System into the trash. The new driver will automatically be favored over the one in ROM because it has a higher version number (1 instead of 0). All my test programs indicate that the new driver works fine in the Mac IIci with System 7, and I would expect it to work fine with System 6 as well. (Thanks to Mike.Alexander@um.cc.umich.edu for figuring out why it didn't work for the boot monitor.)
I. CONTRIBUTORS TO THIS DOCUMENT
Mike.Alexander@um.cc.umich.edu: why you can't replace boot monitor’s driver.
Michael Bach, bach@sun1.ruf.uni-freiburg.de: more on interrupts.
David Brainard, brainard@condor.psych.ucsb.edu, hardware synching.
Thomas Busey: effect of 8•24GC "accel"; Power Mac.
